System and Method  for Imaging and Image Processing

ABSTRACT

One or more objects of interest from a scene are selected. Depth information of the one or more objects is calculated. Additionally, depth information of the scene is calculated. The calculated depth information of the one or more objects is compared with calculated depth information of the scene. Based on the comparison a blur is applied to an image that includes the scene.

The present invention relates to a system and method for creating animage having blurred and non blurred areas using an image capturingdevice. Moreover, the invention relates to an apparatus for creating animage with a low depth of field appearance, to an apparatus for creatingan image with highlighted areas of interest and to an apparatus forcreating an image with highlighted differences in an image sequence.

BACKGROUND OF THE INVENTION

WO 2006/039486 relates to a method for digitally imaging a scene, themethod comprising: using a photo sensor array to simultaneously detectlight from the scene that is passed to different locations on a focalplane; determining the angle of incidence of the light detected at thedifferent locations on the focal plane; and using the determined angleof incidence and the determined depth of field to compute an outputimage in which at least a portion of the image is refocused. ThisInternational application discloses a system as well, comprising: a mainlens; a photo sensor array for capturing a set of light rays; amicrolens array between the main lens and the photo sensor array; a dataprocessor to compute a synthesized refocused image via a virtualredirection of the set of light rays captured by the photo sensor array.

U.S. Pat. No. 7,224,384 relates to an optical imaging system comprising:a taking lens that collects light from a scene being imaged with theoptical imaging system; a 3D camera comprising at least one photosurface that receives light from the taking lens simultaneously from allpoints in the scene and provides data for generating a depth map of thescene responsive to the light; and an imaging camera comprising at leastone photo surface that receives light from the taking lens and providesa picture of the scene responsive to the light; and a light controlsystem that controls an amount of light from the taking lens thatreaches at least one of the 3D camera and the imaging camera withoutaffecting an amount of light that reaches the other of the 3D camera andthe imaging camera.

WO 2008/087652 relates to a method for mapping an object, comprising:illuminating the object with at least two beams of radiation havingdifferent beam characteristics; capturing at least one image of theobject under illumination with each of the at least two beams;processing the at least one image to detect local differences in anintensity of the illumination cast on the object by the at least twobeams; and analyzing the local differences in order to generate athree-dimensional (3D) map of the object.

An object of the present invention is to use information captured by thecamera to blur only selected pixels in the image.

Another object of the present invention is to use depth informationcaptured by the camera and a distance of interest set by an algorithm orby a user to blur only selected pixels.

Another object of the present invention is to use chromatic informationcaptured by the camera and a spectrum of interest set by an algorithm orby a user to blur only selected pixels.

Another object of the present invention is to use difference informationbetween two or more sequential frames to blur only selected pixels.

The term multi aperture digital camera as referred to means a camerathat consists of more than one imaging lenses each having its apertureand lens elements. The term imaging channel refers to a lens and sensorarea of one aperture in a multi aperture digital camera.

Using a multi lens camera allows us to extract distance information ofcertain objects in a scene. The distance between the lenses of thedifferent imaging channels creates a parallax effect causing object thatare not at infinity to appear at different position on the images of thedifferent imaging channels. Calculating these position shifts using analgorithm such as auto-correlation allows us to determine the distanceof each object in the scene. Using a time-of-flight systems allows us tocalculate depth information of objects in a scene by means of emittinglight toward the scene and measuring the time it takes the light to bereturn to the sensor. The farther an object is the longer time it willtake.

Using a structured light system to allow us the calculate depthinformation of objects in a scene is based on a light emitting system inwhich light is emitted in a structured manner such as a grid of dots. Animaging camera is used to image these dots and an algorithm measures toposition of these dots on the its image. The light emitting system andthe imaging camera are separated laterally and therefore a parallaxeffect is present and by calculating the position of the dots or anyother pattern the system can determine the distance of the object inwhich the dot was reflected from.

Using multiple cameras positioned differently allows us to extractdistance information of certain objects in a scene. The distance betweenthe lenses of the different imaging channels creates a parallax effectcausing object that are not at infinity to appear at different positionon the images of the different imaging channels. Calculating theseposition shifts using an algorithm such as auto-correlation allows us todetermine the distance of each object in the scene.

The present inventors found that it possible to blur selected part of animage in order to create a low depth of field appearance and tohighlight certain areas or objects in an image or image sequence. Human,when looking at an image tend to focus the attention to areas that arethe sharpest in their surroundings therefore blurring areas which are oflower interest has a clear advantage.

When using cameras with lenses with a low F/# (focal length divided byaperture diameter) the depth of field becomes smaller when the F/# issmaller. Although this effect may be considered a disadvantage as objectwhich are not positioned at the focus distance are severely blurred itmay also create a 3 dimensional impression of the scene. Using themethod described above for obtaining object distances by calculating thelocal shift between the images of the different imaging channels orusing another technology as described above we can intentionally blurareas in the image that are far from the object of interest which wewant to keep sharp.

The present invention relates to a system and method which may beapplied to a variety of imaging systems. This system and method providehigh quality imaging while considerably reducing the length of thecamera as compared to other systems and methods.

Specifically, the object of the present invention is to provide a systemand a method to improve image capturing devices while. This may beaccomplished by using a 2 or more apertures each using a lens. Each lensforms a small image of the scene. Each lens transfers light emitted orreflected from objects in the scenery onto a proportional area in thedetector. The optical track of each lens is proportional to the segmentof the detector which the emitted or reflected light is projected on.Therefore, when using smaller lenses the area of the detector which theemitted or reflected light is projected on, referred hereinafter as theactive area of the detector, is smaller. When the detector is active foreach lens separately, each initial image formed is significantly smalleras compare to using one lens which forms an entire image. One lenscamera transfers emitter or reflected light onto the entire detectorarea.

According to an embodiment the present invention relates to a method forcreating an image having blurred and non blurred areas using an imagecapturing device capable of depth mapping comprising the steps of:

Selecting one or more object of interest from the scene,

Calculating depth information of said one or more objects of interestfrom the scene,

Retrieving raw data from the multi aperture camera of the completescene,

Calculating depth information of the complete scene,

Comparing the calculated depth information of the selected object ofinterest with the calculated depth of the complete scene.

Applying a blur that is dependent on the result of the comparison.

The step of selecting can be done automatically by an algorithm thatrecognizes area of interest such as faces in conventional photography.Blurring can be achieved by means of convolution of an area of the imagewith a blur filter such as a Gaussian.

More in detail, if a scene consists of a room with 3 people standing at1, 2 and 3 meters from the camera respectively, an object of interestcan be chosen as the person standing at 1 meter. According to theembodiment above, first we will calculate the distance of the object ofinterest and than calculate the distance of all other objects andcompare them. According to this comparison we decide on the type or sizeof blur to apple to each object. In this case a small blur will beapplied to the person standing at 2 meters and a larger blur will beapplied to the person standing at 3 meters. The object of interest whichis the person standing at one meter will not be blurred at all.

The advantage of the embodiment is that a low depth of field appearanceis achieved.

Another advantage is that the selection of object of interest can beapplied automatically or by a user using a touch screen or an inputdevice and a display, in one frame that can be part of a preview modeframe sequence after which a full resolution image may be captured andprocessed to keep the object of interest in focus while blurring otherobject respectively with their distance from the object of interest.This eliminates the need to apply the blur only after the image iscaptured.

According to another embodiment the present invention relates to amethod for creating an image having blurred and non blurred areas usingan image capturing device capable of depth mapping comprising the stepsof:

Capturing an image from the image capturing device,

Calculating a depth map,

Selecting one or more object of interest from the image,

Comparing the calculated depth information of the selected object ofinterest with the calculated depth of the complete scene,

Applying a blur that is dependent on the result of the comparison.

Blurring can be achieved by means of convolution of an area of the imagewith a blur filter such as a Gaussian.

The advantage of this embodiment is that the selection of the object ofinterest is done after the capturing and depth calculating. This allowsthe user to choose different objects of interest or correct hisselection while keeping the non blurred information and depth map.Another advantage is that the selection of objects of interest,comparing with distances of the other objects and blurring accordinglycan be done at a different time with respect to the time of the imagecapturing allowing us the operate these operations on a device differentthan the one used for image capturing. For example the image capturingdevice could be a multi aperture camera integrated in to a mobile phoneor tablet computer and the selection of object of interest and blurringcan be done on a tablet or laptop computer at a different time. Anotheradvantage is that by saving the image and the depth information it ispossible to apply select object of interest and blur multiple time whilesaving the resulting image as a computer file. Each time the selectionof object of interest may be different.

According to another embodiment the present invention relates to amethod for creating an image having blurred and non blurred areas usingan image capturing device, in which the method comprises the followingsteps:

Capturing an image from the image capturing device,

Calculating chromatic properties of objects appearing in the capturedimage,

Selecting one or more object of interest from the image according to thecalculated chromatic properties,

Applying a blur that is dependent on the result of the selection.

Blurring can be achieved by means of convolution of an area of the imagewith a blur filter such as a Gaussian.

The advantage of this embodiment is that we can highlight object withcertain chromatic nature such as tissue suspected as harmful in an imagecaptured by for example an endoscopic camera.

According to another embodiment the present invention relates to amethod for creating an image having blurred and non blurred areas usingan image sequence capturing device, in which the method comprises thefollowing steps:

Capturing an image sequence from the image sequence capturing device,

Calculating differences between the sequential images,

Selecting one or more pixel area of interest from the images accordingto the differences calculated between the sequential frames,

Applying a blur that is dependent on the result of the selection.

Blurring can be achieved by means of convolution of an area of the imagewith a blur filter such as a Gaussian.

The advantage of this embodiment is that objects that are moving orchanging will be highlighted by the effect of the blurring of all otherareas of the image or image sequence.

An example of the embodiment is a surveillance camera coupled with adisplay that is observed by a human. The scene may contain many detailsand objects which make it more difficult for the human to detect movingobjects. By blurring an object that is not moving we attract theattention of the observing human to the moving or changing objects.

The present invention could be integrated in many devices such as adigital camera, digital video camera, mobile phone, a personal computer,tablet, PDA, notebooks, gaming consoles, televisions, monitors,displays, automotive cameras, glasses, helmet, projector, microscopes,imaging endoscopes, imaging medical probe, surveillance systems,inspection systems, speed detection systems, traffic management systems,area access systems, satellite imaging, machine vision and augmentedreality systems.

The invention will be more clearly understood by reference to thefollowing description of preferred embodiments thereof read inconjunction with the figures attached hereto. In the figures, identicalstructures, elements or parts which appear in more than one figure arelabeled with the same numeral in all the figures in which they appear.Dimensions of components and features shown in the figures are chosenfor convenience and clarity of presentation and are not necessarilyshown to scale. The figures are listed below

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a side view of a single lens camera.

FIG. 2 illustrates a sensor array (201) having multiple pixels.

FIG. 3 illustrates a side view of a three lens camera having one sensorand three lenses.

FIG. 4 illustrates an example of a scene as projected on to the sensor.

FIG. 5 illustrates a front view of a three lens camera using onerectangular sensor divided in to three regions.

FIG. 6 illustrates a front view of a three lens camera having onesensor, one large lens and two smaller lenses.

FIG. 7 illustrates a front view of a four lens camera having a onesensor (700) and four lenses.

FIG. 8 illustrates a 16 lens camera having four regions, each containingfour lenses as illustrated in FIG. 7.

DETAILED DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a side view of a single lens camera having a singlelens (102) that can comprise one or more elements and a single sensor(101).

FIG. 2 illustrates a sensor array (201) having multiple pixels where theposition of the green filter, red filter and blue filter are marked by(202), (203) and (204) respectively. The image that will be taken usingthis configuration needs to be processed in order to separate the green,red and blue images.

FIG. 3 illustrates a side view of a three lens camera having one sensor(310) and three lenses (301), (302) and (303). Each one of the said lenswill project the image of the same scene on to segments of the sensormarked by (311), (312) and (313) respectively. Each one of the threelenses will have different color filters integrated within the lens, infront of it or between the lens and sensor (310). Using the describedconfiguration the image acquired by the sensor will be composed of twoor more smaller images, each imaging information from the scene atdifferent spectrums.

FIG. 4 illustrates an example of a scene as projected on to the sensor(401), in each region of the sensor (402), (403) and (404) the samescene is projected but each region will contain information for light atdifferent wavelengths representing different colors according to thefilters integrated within the lens that forms the image on each region.

The described configuration does not require the use of a color mask andtherefore the maximal spatial frequency that can be resolved by thesensor is higher, on the other hand using smaller lens and smalleractive area per channel necessarily means that the focal length of thelens is smaller and therefore the spatial resolution in objects space isdecreased. Overall the maximal resolvable resolution for each colorremains same.

The image acquired by the sensor is composed of two or smaller images,each containing information of the same scene but in different colors.The complete image is then processed and separated in to 3 or moresmaller images and combined together to one large color image.

The Described Method of Imaging has Many Advantages:

-   -   1. Shorter lens track (height) as each one of the lens used is        smaller in size than the single lens covering the same field of        view, the total track (height) of each lens is smaller allowing        the camera to be smaller in height, an important factor for        mobile phone cameras, notebook cameras and other applications        requiring short optical track.    -   2. Reduced Color artifacts—Since each color is captured        separately, artifacts originating from spatial dependency of        each color in a color mask will not appear.    -   3. Lens requirements: each lens does not have to be optimal for        all spectrums used but only for one spectrum, allowing        simplifying the lens design and possibly decreasing the amount        of elements used in each lens as no color correction is needed.    -   4. Larger Depth of Focus: the depth of focus of a system depends        on its focal length. Since we use smaller lenses with smaller        focal lengths, we increase the depth of focus by the scale        factor squared.    -   5. Elimination of focus mechanism: focus mechanisms change the        distance between the lens and the sensor to compensate for the        change in object distance and to assure that the desired        distance is in focus during the exposure time. Such a mechanism        is costly and has many other disadvantages such as:        -   a. Size        -   b. Power consumption        -   c. Shutter lag        -   d. Reliability        -   e. price

Using a fourth lens in addition to the three used for each color red,green and blue (or other colors) with a broad spectral transmission canallow extension of the sensor's dynamic range and improve thesignal-to-noise performance of the camera in low light conditions.

All configuration described above using a fourth lens element can beapplied on other configurations having two or more lenses.

Another configuration that is proposed is using two or more lenses withone sensor having a color mask integrated or on top of the sensor suchas a Bayer mask. In such a configuration no color filter will beintegrated in to each lens channel and all lenses will create a colorimage on the sensor region corresponding to the specific lens. Theresulting image will be processed to form one large image combining thetwo or more color images that are projected on to the sensor.

Three Lens Camera:

Dividing the sensor's active area in to 3 areas, one for each color Red,Green and Blue (or other colors) can be achieved by placing 3 lens onebeside the other as described in the drawing below: The resulting imagewill consist of 3 small images were each contains information of thesame scene in different color. Such a configuration will comprise of 3lenses where the focal length of each lens is 4/9 of an equivalentsingle lens camera that uses a color filter array, these values assume a4:3 aspect ratio sensor.

FIG. 5 illustrates a front view of a three lens camera using onerectangular sensor (500) divided in to three regions (501), (502) and(503). The three lenses (511), (512) and (513) each having differentcolor filters integrated within the lens, in front of the lens orbetween the lens and the sensor are used to form an image of the samescene but in different colors. In This example each region of the sensor(501), (502) and (503) are rectangular having the longer dimension ofthe rectangle perpendicular to the long dimension of the completesensor.

Other three lens configuration can be used, such as using a larger greenfiltered lens and two smaller lenses for blue and red, such aconfiguration will results in higher spatial resolution in the greenchannel since more pixels are being used.

FIG. 6 illustrates a front view of a three lens camera having one sensor(600), one large lens (613) and two smaller lenses (611) and (612). Thelarge lens (613) is used to form an image on the sensor segment marked(603) while the two smaller lenses form an image on the sensor'ssegments marked with (601) and (602) respectively. The larger lens (613)can use a green color filter while the two smaller lenses (611) and(612) can use a blue and red filter respectively. Other color filterscould be used for each lens.

Four Lens Camera:

FIG. 7 illustrates a front view of a four lens camera having a onesensor (700) and four lenses (711), (712), (713) and (714). Each lensforms an image on the corresponding sensor region marked with (701),(702), (703) and (704) respectively. Each one of the lenses will beintegrated with a color filter in side the lens, in front of the lens orbetween the lens and the sensor. All four lenses could be integratedwith different color filter or alternatively two of the four lensescould have the same color filter integrated in side the lens, in frontof the lens or between the lens and the sensor. For example using twogreen filters one blue filter and one red filter will allow more lightcollection in the green spectrum.

M×N Lens Camera:

Using M and/or N larger than 2 allows higher shortening factor andhigher increase in depth of focus.

FIG. 8 illustrates a 16 lens camera having 4 regions (801), (802), (803)and (804) each containing four lenses as illustrated in FIG. 7.

1-15. (canceled)
 16. A method for creating an image comprising:selecting one or more objects of interest from a scene; calculatingfirst depth information of the one or more objects of interest;calculating second depth information of the scene; comparing the firstdepth information with the second depth information; and creating animage having at least one blurred area and at least one non-blurred areabased on the comparison.
 17. The method of claim 16, wherein the firstdepth information is calculated using a multi aperture digital camerahaving a plurality of imaging channels.
 18. The method of claim 17,wherein the plurality of imaging channels includes filters withidentical chromatic transmission properties.
 19. The method of claim 17,wherein the plurality of imaging channels each includes a filter withproportional chromatic transmission properties.
 20. The method of claim17, wherein the first depth information is calculated by comparing aplurality of respective images from the plurality of imaging channels.21. The method of claim 16, wherein the first depth information iscalculated using a time-of-flight system.
 22. The method of claim 16,wherein the first depth information is calculated by comparing two ormore images captured by a differently positioned digital camera.
 23. Themethod of claim 16, wherein the image having at least one blurred areaand at least one non-blurred area has a low depth of field appearance.24. A method for creating an image having blurred and non blurred areas,the method comprising: capturing an image; calculating a depth map;selecting one or more objects of interest from the image; comparing thecalculated depth map with depth information of the selected one or moreobjects; and applying a blur to the image based on the comparison. 25.The method of claim 24, wherein responsive to applying the blur, theimage has a low depth of field appearance.
 26. A method for creating animage having blurred and non blurred areas, the method comprising:capturing an image sequence comprising sequential images; calculatingdifferences between the sequential images; selecting one or more pixelareas of interest from the sequential images based on the calculateddifferences; and applying a blur to the image sequence based on theselection of the one or more pixel areas.
 27. The method of claim 26,wherein responsive to applying the blur, the differences between thesequential images are highlighted in the image sequence.